12 research outputs found

    Evaluating HeLI with non-linear mappings

    Get PDF
    Peer reviewe

    Language Set Identification in Noisy Synthetic Multilingual Documents

    Get PDF
    Proceeding volume: Part IIn this paper, we reconsider the problem of language identification of multilingual documents. Automated language identification algorithms have been improving steadily from the seventies until recent years. The current state-of-the-art language identifiers are quite efficient even with only a few characters and this gives us enough reason to again evaluate the possibility to use existing language identifiers for monolingual text to detect the language set of a multilingual document. We are using a previously developed language identifier for monolingual documents with the multilingual documents from the WikipediaMulti dataset published in a recent study. Our method outperforms previous methods tested with the same data, achieving an F 1-score of 97.6 when classifying between 44 languages.Peer reviewe

    HeLI, a Word-Based Backoff Method for Language Identification

    Get PDF
    In this paper we describe the Helsinki language identification method, HeLI, and the resources we created for and used in the 3rd edition of the Discriminating between Similar Languages (DSL) shared task, which was organized as part of the VarDial 2016 workshop. The shared task comprised of a total of 8 tracks, of which we participated in 7. The shared task had a record number of participants, with 17 teams providing results for the closed track of the test set A. Our system reached the 2nd position in 4 tracks (A closed and open, B1 open and B2 open) and in this paper we are focusing on the methods and data used for those tracks. We describe our word-based back-off method in mathematical notation. We also describe how we selected the corpus we used in the open tracks.Peer reviewe

    HeLI-based Experiments in Swiss German Dialect Identification

    Get PDF
    Peer reviewe
    corecore